Image pickup apparatus, information processing system, mat, and image generation method

ABSTRACT

An image of a block or the like entering a space above a play mat is picked up by an image pickup apparatus, and an information processing apparatus detects and recognizes the block on the basis of a picture of the block in a picked up image and performs information processing. The play mat includes a play field that defines a detection space for the block, and a calibration chart including at least a plurality of regions of colors of different luminances. The image pickup apparatus adjusts an exposure time period, a gain value for each color component and a correction rule upon gamma correction on the basis of the picture of the calibration chart in the picked up image.

TECHNICAL FIELD

The present invention relates an information processing system thatperforms an information process including recognition of a real objectin which a pickup image is used, an image pickup apparatus and a matincluded in the system and an image production method used in theinformation processing system.

BACKGROUND ART

Conventionally, a toy wherein a three-dimensional object can beassembled by connecting a plurality of blocks or parts to each other isknown. For example, blocks that can be freely assembled in response toan idea of a user and individually have a basic shape such as a cube ora rectangular solid, blocks or parts for exclusive use formed so as toassemble a three-dimensional object supposed in advance in accordancewith a design drawing and so forth are widely spread. Further, atechnology wherein a computer recognizes a movement or a shape variationof an object in a real space and performs some information processing isexpected to be applied to a toy, a teaching material for learning and soforth (for example, refer to PTL 1 and NPL 1).

CITATION LIST Patent Literature

-   [PTL 1] Japanese Patent Laid-Open No. 2008-73256

Non Patent Literature

-   [NPL 1] Posey: Instrumenting a Poseable Hub and Strut Construction    Toy, Michael Philetus Weller, Ellen Yi-Luen Do, Mark D Gross,    Proceedings of the Second International Conference on Tangible and    Embedded Interaction, 2008, pp 39-46

SUMMARY Technical Problem

When an object whose image is picked up by a camera is to be recognizedin such a technology as described above, a color of the object isimportant information. However, the color of a picture of the object inthe picked up image varies depending upon an image pickup environmentsuch as an illumination, a material of the object and so forth andcorrect recognition is sometimes obstructed. For example, even if thesame object is used, this is recognized as a different object in somecases, or such a case occurs in which a subtle difference of colorsometimes can be distinguished or sometimes cannot be distinguished. Inthis manner, the recognition accuracy becomes instable, and it issupposed that a trouble may occur in processing at a later stage.

The present invention has been made in view of such a subject asdescribed above, and it is an object of the present invention to providea technology by which information processing including recognition of areal object in which a picked up image is used can be implemented instability and with high accuracy.

Solution to Problem

In order to solve the subject described above, an aspect of the presentinvention relates to an image pickup apparatus. This image pickupapparatus is an image pickup apparatus that outputs data of a picked upimage to be used for recognition of a real object, including an outputimage production unit configured to produce data of a picked up image tobe outputted from data of a non-processed image picked up using an imagepickup element, an adjustment processing unit configured to evaluate thepicked up image and adjust at least one of an image pickup condition inthe image pickup element and a processing condition in the output imageproduction unit, and a data sending out unit configured to output thedata of the picked up image to an apparatus that recognizes the realobject, wherein the adjustment processing unit adjusts the image pickupcondition and the processing condition based on color information of apicture of a calibration chart that is provided in a space of arecognition target and is configured from a plurality of color regionshaving luminances different from each other.

Another aspect of the present invention relates to an informationprocessing system. This information processing system is an informationprocessing system including an image pickup apparatus that outputs dataof a picked up image obtained by image pickup of a real object and aninformation processing apparatus that recognizes the real object usingthe picked up image, wherein the image pickup apparatus includes anoutput image production unit configured to produce data of a picked upimage to be outputted from data of a non-processed image picked up usingan image pickup element, an adjustment processing unit configured toevaluate the picked up image and adjust at least one of an image pickupcondition in the image pickup element and a processing condition in theoutput image production unit, and a data sending out unit configured tooutput the data of the picked up image to the information processingapparatus, and the adjustment processing unit adjusts the image pickupcondition and the processing condition based on color information of apicture of a calibration chart that is provided in a space of arecognition target and is configured from a plurality of color regionshaving luminances different from each other, and the informationprocessing apparatus recognizes a block by comparison between colorinformation of the picture of the real object in the picked up image andcolor information of real objects registered already with each other andperforms a process set for the block.

A further aspect of the present invention relates to a mat. This mat isa mat that is placed in an image pickup space in an informationprocessing system that recognizes a real object based on colorinformation of a picture in a picked up image, including a play fieldthat defines a bottom region of a space in which, when the real objectis placed into the space, the real object is determined as a recognitiontarget, and a calibration chart configured from a plurality of colorregions having different luminances from each other and used by an imagepickup apparatus to adjust, based on color information of a picture inthe picked up image, at least one of an image pickup condition and aprocessing condition when data of the picked up image to be outputted isproduced.

A still further aspect of the present invention relates an imageproduction method. This image production method is an image productionmethod, by an image pickup apparatus, of producing data of a picked upimage to be used for recognition of a real object, including a step ofproducing data of a picked up image to be outputted from data of anon-processed image picked up using an image pickup element, a step ofevaluating the picked up image and adjusting at least one of an imagepickup condition in the image pickup element and a processing conditionin the producing step, and a step of outputting the data of the pickedup image to an apparatus that recognizes the real object, wherein thestep of adjusting adjusts the image pickup condition and the processingcondition based on color information of a picture of a calibration chartthat is provided in a space of a recognition target and is configuredfrom a plurality of color regions having luminances different from eachother.

It is to be noted that arbitrary combinations of the componentsdescribed above and conversions of the representation of the presentinvention between a method, an apparatus, a system, a recording medium,a computer program and so forth are effective as modes of the presentinvention.

Advantageous Effect of Invention

According to the present invention, information processing includingrecognition of a real object in which a picked up image is used can beperformed stably and with high accuracy.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a view depicting an example of a configuration of aninformation processing system to which an embodiment can be applied.

FIG. 2 is a view depicting a configuration of an internal circuit of aninformation processing apparatus in the present embodiment.

FIG. 3 is a view depicting a configuration of functional blocks of theinformation processing apparatus in the present embodiment.

FIG. 4 is a view depicting a configuration of functional blocks of animage pickup apparatus in the present embodiment.

FIG. 5 is a flow chart illustrating a procedure of informationprocessing performed by the image pickup apparatus and the informationprocessing apparatus in the present embodiment and including blockrecognition.

FIG. 6 is a view exemplifying an upper face of a play mat in the presentembodiment.

FIG. 7 is a view depicting a modification to a calibration chart in thepresent embodiment.

FIG. 8 is a flow chart illustrating a processing procedure of the imagepickup apparatus that adjusts an exposure time period, a gain value anda correction rule upon gamma correction at S12 of FIG. 5.

FIG. 9 is a flow chart illustrating a processing procedure of the imagepickup apparatus that adjusts an exposure time period, a gain value anda correction rule upon gamma correction at S12 of FIG. 5.

FIG. 10 is a view illustrating white balance adjustment performed in thepresent embodiment.

FIG. 11 is a view illustrating a variation of a pixel value when thewhite balance is adjusted for a plurality of regions configuring thecalibration chart in the present embodiment.

FIG. 12 is a view illustrating a process for adjusting an exposure timeperiod and a gain value at S32 to S38 of FIG. 8.

FIG. 13 is a view illustrating a process for adjusting a correction rulein a gamma correction unit at S44 and S46 of FIG. 9.

FIG. 14 is a view illustrating a technique for adjusting a correctioncurve on the basis of a color distribution of blocks in the presentembodiment.

FIG. 15 is a view illustrating a method of adjusting a gain value forRGB utilizing a color chart in the present embodiment.

FIG. 16 is a view depicting a modification to the play mat in thepresent embodiment.

DESCRIPTION OF EMBODIMENT

FIG. 1 depicts an example of a configuration of an informationprocessing system to which an embodiment can be applied. An informationprocessing system 1 includes blocks 20 a, 20 b and 20 c placed on a playmat 18, an image pickup apparatus 12 for picking up an image of thespace above the play mat 18, an information processing apparatus 10 forrecognizing the blocks on the basis of a picked up image and performinginformation processing in response to a result of the recognition, aninputting apparatus 14 for accepting a user operation to the informationprocessing apparatus 10, and a display apparatus 16 for outputting aresult of the information processing as an image.

The play mat 18 includes a play field 100 that is a region in which theblocks 20 a, 20 b and 20 c of a recognition target are to be placed, anda calibration chart 102 for determining an image pickup condition or anadjustment or correction condition for the image pickup apparatus 12.The material of the play mat 18 is any of paper, a plate, a cloth, vinyland so forth and is not limited especially. Also the shape of the playmat 18 is not limited especially and may be such a quadrangular shape asdepicted in FIG. 1 or may be any other polygon or a shape surrounded bya curve like a circle or an ellipsis. The play mat 18 may have someother application as a top plate of a desk, a game board or the like.

Although the play field 100 or the calibration chart 102 typically is aregion or a graphic printed on the play mat 18, it may otherwise beconfigured such that an article prepared separately is attached to orplaced on a plane that configures the play mat 18. Although each of theblocks 20 a, 20 b and 20 c most simply is such a general “block” of acube or a rectangular parallelepiped, it is not limited in shape orfunction only if it is an object existing in the real space.

For example, each block may be an article of a more complicated shapesuch as a miniature of an article existing in the real world such as adoll or a minicar or a part of such a miniature as just mentioned or apiece of a game, and also the size, material, color, quantity to be usedand so forth of such blocks are not limited. Further, each block may bestructured such that it can be assembled and disassembled by a user ormay be a finished product. Further, each block may have a communicationmechanism or a computer for establishing communication with theinputting apparatus 14 or the information processing apparatus 10, amechanism for driving the inputting apparatus 14 in response to a useroperation or under the control of the information processing apparatus10 and so forth or may not have any of such mechanisms.

The image pickup apparatus 12 is a video camera having an image pickupelement of a charge coupled device (CCD) type, a complementary metaloxide semiconductor (CMOS) type or the like and picks up an image of thespace above the play mat 18 and supplies the image to the informationprocessing apparatus 10. The image pickup apparatus 12 may be amonocular camera as depicted in FIG. 1 or may be a stereo cameraconfigured from two cameras for picking up an image of the same spacefrom left and right positions having a known distance therebetween.

The display apparatus 16 may be a general display unit such as a liquidcrystal display unit, a plasma display unit or an organicelectroluminescence (EL) display unit. Alternatively, a television setin which the display apparatus 16 and a speaker not depicted areprovided integrally may be used. The inputting apparatus 14 accepts,when it is operated by a user, a request for starting of processing,ending, selection of a function or inputting of various commands andsupplies an accepted request as an electric signal to the informationprocessing apparatus 10. The inputting apparatus 14 may be any ofgeneral inputting apparatus such as a game controller, a keyboard, amouse, a joystick, a touch pad that is provided on the screen of thedisplay apparatus 16 and so forth or may be any combination of them.

The information processing apparatus 10 may be, for example, a gameapparatus or a personal computer, and an information processing functionmay be implemented by loading a necessary application program into theinformation processing apparatus 10. The information processingapparatus 10 detects or chases the block 20 a, 20 b or 20 c on the playfield 100. To this end, the information processing apparatus 10 acquiresa moving image picked up by the image pickup apparatus 12 at apredetermined frame rate and extracts and recognizes a picture of theblock from the image. Thereafter, if the user moves the block by a handor by the inputting apparatus 14 or the information processing apparatus10 itself controls the movement of the block, then the informationprocessing apparatus 10 chases the variation of the position. If theshape of the block is variable, then also the shape variation is chased.

Although the information processing apparatus 10 may perform someinformation processing utilizing a detection result or a chase result ofa block, the contents of the information processing are not limitedespecially. For example, a moving image that is being picked up by theimage pickup apparatus 12 and in which an additional object to a pictureof a block is rendered or a block is replaced by a corresponding objectmay be displayed on the display apparatus 16. Alternatively, a game maybe progressed in response to the position of a block, and a game screenimage representative of the contents of the game may be displayed on thedisplay apparatus 16 or effect sound or voice may be outputted from aspeaker not depicted.

Further, it can be recognized by those skilled in the art that variousprocesses may be performed using a detection result or a chase result ofa block. In the following, a technique for performing detection of ablock with a high degree of accuracy is described especially focusing ona technique for improving the recognition accuracy of a block byacquiring color information of the block with a high degree of accuracy.

It is to be noted that connection between the information processingapparatus 10 and the image pickup apparatus 12, inputting apparatus 14and display apparatus 16 may be established by various networksirrespective of whether they are wire networks or wireless networks.Alternatively two or all of them may be combined and providedintegrally. Depending upon a process to be performed at a succeedingstep by the information processing apparatus 10, a speaker or amicrophone may be connected further, or the inputting apparatus 14 orthe display apparatus 16 may not be provided. In this manner, aninputting apparatus other than the image pickup apparatus 12 and anoutputting apparatus can be combined suitably in the informationprocessing system 1.

FIG. 2 depicts a configuration of an internal circuit of the informationprocessing apparatus 10. The information processing apparatus 10includes a central processing unit (CPU) 22, a graphics processing unit(GPU) 24 and a main memory 26. The CPU 22 controls processing or signaltransmission in or between internal components of the informationprocessing apparatus 10 on the basis of a program such as an operatingsystem or an application. The GPU 24 performs image processing. The mainmemory 26 is configured from a random access memory (RAM) and storesprograms or data necessary for processing.

The components mentioned are connected to each other by a bus 30.Further, an input/output interface 28 is connected to the bus 30. To theinput/output interface 28, a communication unit 32 configured from aperipheral apparatus interface of universal serial bus (USB), Instituteof Electrical and Electronics Engineers (IEEE)1394 or the like or anetwork interface for a wire or wireless local area network (LAN), astorage unit 34 such as a hard disk drive or a nonvolatile memory, anoutputting unit 36 for outputting data to an outputting apparatus suchas the display apparatus 16 or a speaker, an inputting unit 38 forinputting data from the image pickup apparatus 12 or the inputtingapparatus 14, and a recording medium driving unit 40 for driving aremovable recording medium such as a magnetic disk, an optical disc or asemiconductor memory.

The CPU 22 executes the operating system stored in the storage unit 34to control the entire information processing apparatus 10. The CPU 22further executes various programs read out from a removable recordingmedium and loaded into the main memory 26 or downloaded through thecommunication unit 32. The GPU 24 has a function of a geometry engineand a function of a rendering processor, and performs a renderingprocess in accordance with a rendering instruction from the CPU 22 andstores a resulting display image into a frame buffer not depicted. Then,the GPU 24 converts the display image stored in the frame buffer into avideo signal and outputs the video signal to the outputting unit 36.

FIG. 3 depicts a configuration of functional blocks of the informationprocessing apparatus 10. The functional blocks depicted in FIG. 3 andFIG. 4 hereinafter described can be configured, in hardware, from such aCPU, a GPU, a main memory, an outputting unit, an inputting unit, amicrocomputer, various arithmetic operation units, a buffer memory andso forth as depicted in FIG. 2, and is implemented, in software, from acomputer program or the like loaded from a hard disk or a recordingmedium into the main memory. Accordingly, it is recognized by thoseskilled in the art that the functional blocks described above can beimplemented in various forms from only hardware, from only software orfrom a combination of hardware and software and are not limited to anyof them.

The information processing apparatus 10 includes an input informationacquisition unit 52 that acquires information relating to a useroperation accepted by the inputting apparatus 14, a picked up imageacquisition unit 54 that acquires data of a picked up image from theimage pickup apparatus 12, a detection unit 60 that performs detectionof a block, a recognition unit 62 that recognizes a detected block, anoutput data production unit 70 that produces output data in response toa position or a movement of the block, an image data storage unit 72that stores image data to be used in processing such as the picked upimage or a depth image, and a block database 64 that stores basicinformation of the block.

It is to be noted that the information processing apparatus 10 mayfurther include a function block according to contents of informationprocessing performed utilizing a state of a block suitably such as afunction block that controls a driving mechanism in the inside of theblock. The input information acquisition unit 52 notifies the picked upimage acquisition unit 54 and the output data production unit 70 ofcontents of an operation performed by the user through the inputtingapparatus 14. This operation includes starting and ending requests ofinformation processing of a game or the like, command inputting during aprocess and so forth. The starting/ending request of a process isnotified of also to the image pickup apparatus 12 through the picked upimage acquisition unit 54 to control starting/ending of moving imagepickup by the image pickup apparatus 12.

The picked up image acquisition unit 54 acquires frame data of a movingimage obtained as a result of image pickup at a predetermined framerate. The predetermined rate here may be a frame rate of a moving imageto be picked up by the image pickup apparatus 12 or may be a rate lowerthan the frame rate. The picked up image acquisition unit 54 storesacquired data of a picked up image into the image data storage unit 72.

The detection unit 60 reads out data of a picked up image from the imagedata storage unit 72 and detects or chase a block on the play field 100on the basis of a picture on the image. For example, the detection unit60 recognizes that a block is placed on the basis of the differencebetween a picked up image in a state in which a block is not placed andframe images in a moving image being picked up. If it is assumed thatthe color of the play field 100 is known, then also it is possible todetect a picture having a different color from the known color as ablock. After a block is detected once, a movement of the block can bechased by comparison with a preceding frame image. Various technologieshave been proposed for chasing a movement of an object in an image, andany of the technologies may be adopted in the present embodiment.

Alternatively, the image pickup apparatus 12 may be configured from astereo camera such that a block is detected from depth information fromthe cameras in an image pickup space. In particular, from depthinformation of image pickup objects obtained by the stereo camera, theposition or the height of the image pickup objects in athree-dimensional space having a bottom face given by the play field 100by coordinate transformation. Then, if an object whose height from theplane of the play field 100 exceeds a predetermined threshold valueexists, then this is detected as a block. A technique for acquiringdepth information of an image pickup object using a stereo camera iswidely known. It is to be noted that, in the present mode, especiallythe play field 100 defines the bottom of the three-dimensional space ofa detection target and can determine a block entering the space abovethe bottom of the three-dimensional space as a detection targetirrespective of whether or not the block is placed on the play field100.

If a block is detected, then the detection unit 60 notifies therecognition unit 62 of this together with position information of thepicture of the block on the image. Further, the detection unit 60successively supplies results of the chase of the detected block,namely, position information in each frame, to the output dataproduction unit 70 until the block goes out of the play field 100. Therecognition unit 62 acquires characteristics of an appearance of theblock existing at the position notified of from the detection unit 60using the picked up image read out from the image data storage unit 72.

Then, the recognition unit 62 compares the appearance characteristic ofthe block with characteristics of blocks stored in advance in the blockdatabase 64 to specify to which block the picture detected in the imagebelongs. In the block database 64, information that associatesidentification numbers of blocks, characteristics in appearance such asa color or a shape, data to be outputted in response to the position orthe movement of a block and so forth with each other is stored inadvance. The recognition unit 62 notifies the output data productionunit 70 of an identification number of the block obtained in such amanner as described above and the position information of the block inthe image in an associated relationship with each other.

The output data production unit 70 produces data of an image to beoutputted in response to the position and the movement of the block andoutputs the data to the display apparatus 16. Therefore, the output dataproduction unit 70 continues to acquire position information of theblock from the detection unit 60 and refers to the block database 64 onthe basis of the identification number of the block notified of from therecognition unit 62 to determine a process to be performed. For example,the output data production unit 70 renders an object associated with theblock at the position of the block on the picked up image or performsprocessing for the picked up image in response to a positionalrelationship of the block to a different block.

In such a case as described above, the output data production unit 70reads out the picked up image from the image data storage unit 72 andcarries out a corresponding process. However, particular contents of theinformation processing are not limited as described above, and theinformation processing may not necessarily use the picked up image.Further, the output data production unit 70 may produce not only data ofan image but also audio data of effective sound or the like. In theblock database 64, a program that defines a process to be performed inresponse to data or a movement of an image to be rendered at theposition of the block in the picked up image and so forth are stored inan associated relationship with an identification number of the block.

In this manner, in the present embodiment, a block placed on the playfield 100 is recognized on the basis of a picked up image by the imagepickup apparatus 12 and performs corresponding information processing.Upon recognition, color information is significant together with a shapeand a size of the block. On the other hand, despite being the sameblock, a block sometimes changes in color of a picture on a picked upimage from various factors such as an illumination environment in a realspace or presence or absence or a color of an article existing aroundthe block. Consequently, the block is sometimes recognized as adifferent block or a plurality of blocks having similar colors createconfusion, and as a result, it is considered that information processingis not performed appropriately. Therefore, in the present embodiment,the calibration chart 102 is provided on the play mat 18 such thatadjustment or correction of an image pickup condition or obtained imagedata is performed in order to improve the recognition accuracy of ablock.

FIG. 4 depicts a configuration of function blocks of the image pickupapparatus 12. The image pickup apparatus 12 includes an image pickupunit 80, a gain processing unit 84, an adjustment processing unit 86, ademosaic unit 88, a gamma correction unit 90 and a data sending out unit92. It is to be noted that, where the image pickup apparatus 12 isconfigured as a stereo camera, the function blocks other than the datasending out unit 92 are provided for each of cameras of the stereocamera. The image pickup unit 80 is configured from an image pickupelement such as a CCD element or a CMOS element and outputs an exposureresult at a predetermined timing (for example, 60 times/second) as pixelvalues of images. The image outputted here is a RAW image (non-processedimage) configured from one of red, green and blue (RGB) colors and isoutputted in a predetermined pixel order such as a raster order to thegain processing unit 84.

The gain processing unit 84 is configured from a variable gainamplification circuit or the like, and amplifies a red primary colorsignal (R value), a green primary color signal (G value) and a blueprimary color signal (B value) outputted from the image pickup unit 80with set gain values and then outputs resulting signals in a rasterorder or the like to the demosaic unit 88. The adjustment processingunit 86 determines, on the basis of a picture of the calibration chart102 on the picked up image, at least one of an exposure time period ofthe image pickup unit 80, a gain value to be set to the gain processingunit 84 and a correction rule for pixel values in the gamma correctionunit 90. As hereinafter described, they may be determined on the basisnot only of the calibration chart 102 but also of an actual picture ofthe block. Results of the determination are each notified of to theimage pickup unit 80, gain processing unit 84 and gamma correction unit90.

The demosaic unit 88 acquires data of RAW images whose pixel values areamplified from the gain processing unit 84 and executes a demosaic(de-mosaic) process of complementing color information to the pixels onthe basis of their peripheral pixels to create a full color image. Tothis demosaic process, a well-known technology can be applied. Forexample, for a pixel having only a G value in the RAW images, RGB valuesare determined such that, as an R value, an average of R values ofpixels neighboring on the left and the right is used; as a G value, theG value of the pixel is used as it is; and as a B value, a B value of apixel neighboring upwardly or downwardly is used. By performing asimilar complementing process for all pixels, an RGB image in which eachpixel has RGB values is obtained. The demosaic unit 88 furthersubstitutes the RGB image into a predetermined conversion formula toobtain a YCbCr image.

The gamma correction unit 90 corrects a full color image produced insuch a manner as described above with a predetermined correction formulato produce data of an output image. The gamma correction unit 90 isbasically configured so as to convert a pixel value in accordance with awell-known gamma curve such that color signals output by the imagepickup elements are displayed in an appropriate color balance on adisplay apparatus. On the other hand, when a block is to be detected, animage corrected in accordance with a curve obtained by deforming a gammacurve is used such that similar colors can be distinguished from eachother with a high degree of accuracy.

The gamma correction unit 90 outputs data of a produced output image tothe adjustment processing unit 86 and the data sending out unit 92. Thedata sending out unit 92 transmits the data of the output image to theinformation processing apparatus 10 by general communication means. Theadjustment processing unit 86 acquires, every time various parametersare adjusted in such an order as hereinafter described, an output imageafter the adjustment from the gamma correction unit 90 and evaluates theoutput image to perform further adjustment or to determine to end theadjustment.

Now, operation of the information processing system 1 implemented by theconfiguration described above is described. FIG. 5 is a flow chartillustrating a procedure of information processing including blockrecognition, which is performed by the image pickup apparatus 12 and theinformation processing apparatus 10. Operation of this flow chart isstarted when a user issues a request for starting processing to theinformation processing apparatus 10 through the inputting apparatus 14.Note that it is assumed that the image pickup apparatus 12 is set by theuser such that the entire play mat 18 is included in the field of view.

Further, if the picked up image acquisition unit 54 of the informationprocessing apparatus 10 issues a request to start image pickup to theimage pickup apparatus 12, then the image pickup unit 80 of the imagepickup apparatus 12 starts image pickup (S10). In this picked up image,the calibration chart 102 is shown. As occasion demands, the field ofview of the image pickup apparatus 12 may be optimized such that thecalibration chart 102 is shown well. For example, the informationprocessing apparatus 10 instructs the user to manually perform directionadjustment or magnification adjustment through the display apparatus 16or the like. Alternatively, the information processing apparatus 10 maycontrol the field of view of the image pickup apparatus 12 utilizing apan-tilter or an electronic zoom not depicted provided in the imagepickup apparatus 12.

In the processes described above, the information processing apparatus10 acquires a picked up image once and detects a region in which apicture of the calibration chart 102 exists in the image. Theinformation processing apparatus 10 may notify the image pickupapparatus 12 of a result of the detection such that the image pickupapparatus 12 recognizes the calibration chart 102. On the basis of thepicked up image of the calibration chart 102 obtained in this manner,the adjustment processing unit 86 of the image pickup apparatus 12adjusts at least one of an exposure time period by the image pickup unit80, a gain value to be set to the gain processing unit 84 and acorrection rule for the gamma correction unit 90 (S12).

By this adjustment, an output image in which, even if an image pickupenvironment changes, this does not have a significant influence on thecolor of the picture of the block is obtained. Further, an output imagein which a small difference between similar colors is emphasized isobtained. The data sending out unit 92 of the image pickup apparatus 12outputs data of the output image after the adjustment to the informationprocessing apparatus 10. The detection unit 60 of the informationprocessing apparatus 10 supervises, on the basis of the picked up imagetransmitted thereto from the image pickup apparatus 12, whether or not anew block enters the space above the play field 100 (S14).

If a new block enters (Y at S14), then the recognition unit 62 acquirescolor information of the block on the image (S16). Further, therecognition unit 62 specifies a region in which the color continues as aregion of the picture of the block and acquires a shape of the region(S18). Then, the recognition unit 62 refers to the block database 64 tospecify an identification number assigned to the block and associatesthe identification number and the picture on the image with each otheron basis of the color and the shape of the region (S20). Although theidentification number here is assigned upon shipment or the like of theblock, this is not limited to a number but may be a name or the like.

It is to be noted that, depending upon targeted information processing,such a mode may be available that, after the user places the block onthe play field 100, the block is registered on the spot by selecting, ordesignating by voice, a name to be applied to the block or an object tobe associated with the block from within a list displayed on the displayapparatus 16. In this case, for example, characteristics of anappearance such as a color of the block, a name designated by the userand so forth are stored in an associated relationship with each otherinto the block database 64 such that the registration information may bemaintained also at a later timing.

Also in this mode, registration information in the past can be specifiedaccurately by preventing the color of the picture of the block frombeing changed by an external factor. It is to be noted that, in thiscase, a program for a process or the like to be performed in response toimage data or a movement of an object may be stored in the blockdatabase 64 in an associated relationship with a name or the like thatcan be designated by the user in place of the block. Meanwhile, thedetection unit 60 chases the picture of the block detected dependingupon Y at S14 and continuously notifies the output data production unit70 of the position information of the picture (S22).

The output data production unit 70 performs information processing inaccordance with the chase result on the basis of the identificationnumber of the block notified of from the recognition unit 62 to produceoutput data and outputs the output data from the display apparatus 16 orthe like (S24). While no block exists in the space above the play field100 at S14 (N at S14), an initial screen image of a game or the like,the picked up image or the like continues to be displayed (S24). Whilethe user does not request to end the process through the inputtingapparatus 14 (N at S26), if a new block enters the space above the playfield 100 (Y at S14), then the picture and the identification number ofthe block are associated with each other and the picture is added as achase target (S16 to S22).

If a new block does not enter (N at S14), then only the block that hasbeen detected till then continues to be chased and output data isproduced and outputted in response to a result of the chase or the like(S24). If the user requests to end the process through the inputtingapparatus 14, then all processes including image pickup are ended (Y atS26). It is to be noted that the picked up image used in detection andrecognition of a block at S14 to S20 is an image whose variousparameters have been adjusted by the adjustment processing unit 86 ofthe image pickup apparatus 12 in order to recognize the blockaccurately. On the other hand, in order to chase the picture of theblock or produce output data at S24, an image for which at least part ofsuch adjustment has not been performed may be used.

Especially, where a picked up image is processed to produce a displayimage, it is more natural to use an image of a state proximate to anactual state. Also in the chase process, where a technique in whichcolor information of a block is not significant like a case in whichonly a contour of a block is chased from a preceding frame is used, apicked up image not adjusted as yet can be used. Therefore, the imagepickup apparatus 12 may transmit both of data of an image for whichadjustment is performed and another image for which no adjustment isperformed to the information processing apparatus 10 as occasiondemands.

For example, in a mode in which a new block is detected in parallel tochase of a block or production of an output image, both data aretransmitted at the same time and are selectively used in the inside ofthe information processing apparatus 10. In another mode in which allblocks are recognized as an initial process and a new block does notenter at the stages of chase of a block and production of an outputimage, an adjusted picked up image is transmitted in an initial process,and changeover is performed such that, upon later operation, a picked upimage that is not adjusted as yet is transmitted. At this time, theinformation processing apparatus 10 notifies the image pickup apparatus12 of a timing of the changeover.

Further, in the example of FIG. 5, adjustment of various parameters inthe image pickup apparatus is performed only at S12 using thecalibration chart 102. On the other hand, at least part of parametersmay be adjusted in response to a distribution of colors of blocks thathave actually entered the space above the play field 100 or of blocksregistered by the user as hereinafter described. In this case, everytime a new block is detected, color information acquired at S16 isnotified of from the information processing apparatus 10 to the imagepickup apparatus 12 such that the adjustment processing unit 86 of theimage pickup apparatus 12 successively records distribution informationof the colors of the blocks and performs adjustment as occasion demands.

FIG. 6 exemplifies a top plan view of the play mat 18. As describedhereinabove, the play mat 18 includes a play field 100 indicating aregion in which a user places or moves a block, and a calibration chart102 for optimizing an image pickup condition or adjustment andcorrection conditions of the image. The play field 100 preferably is aregion filled with a color different from that of a block such that onlythe picture of the block can be extracted with high accuracy on theimage. However, it is not the gist that the present embodiment islimited to this, and a pattern, a sketch or the like may be drawn in theplay field 100. Also the shape of the play field 100 is not limited tothat depicted in FIG. 6.

The calibration chart 102 is configured from a plurality of achromaticcolor regions of different luminances as depicted in an enlarged scaleon the lower side of the play mat 18 in FIG. 6. In the case of FIG. 6,the calibration chart 102 is configured from four regions of a whiteregion 104, a black region 106, a first grey region 108 and a secondgrey region 110. However, the disposition or size of the regions is notlimited especially. Also the number of grey regions, namely, the numberof stages of intermediate luminances is not limited especially. Further,as hereinafter described, a region of a chromatic color may be providedin the calibration chart 102.

The adjustment processing unit 86 extracts, if it acquires data aftergamma correction of an image obtained by image pickup of the calibrationchart 102 from the gamma correction unit 90, pictures of the whiteregion 104, black region 106 and first grey region 108 (or second greyregion 110) and adjusts the exposure time period and the gain value ofeach of the regions at least with the following policies.

(1) To cause the luminance to have a predetermined target value(2) To adjust balances of RGB to each other (white balance adjustment)

Further, the adjustment processing unit 86 compares the luminances of atleast two regions from among the white region 104, black region 106,first grey region 108 and second grey region 110 and adjusts a gainvalue and a correction rule upon gamma correction at least with thefollowing policy.

(3) To make it possible to obtain the difference in luminanceappropriately.

FIG. 7 depicts a modification to the calibration chart 102. Thecalibration chart of the present example is configured from a grey stepregion 112 configured from nine regions of achromatic colors ofdifferent luminances from black to white, and a grey region 114 having apredetermined luminance and having a comparatively great area. Thecalibration chart 102 is used to determine an area and a stage numberoptimum to perform adjustment with high accuracy in response to a colordistribution of blocks which can be supposed. The policy for adjustmentis similar to that described hereinabove.

FIGS. 8 and 9 are flow charts illustrating a processing procedure by theimage pickup apparatus 12 for adjusting the exposure time period, gainvalue and correction rule upon gamma correction at S12 of FIG. 5. RAWimages acquired by the image pickup unit 80 and including a picture ofthe calibration chart 102 as described hereinabove become an outputimage through processes by the gain processing unit 84, demosaic unit 88and gamma correction unit 90. The gamma correction unit 90 supplies dataof the image to the adjustment processing unit 86. Note that it isassumed that, although omitted in the depicted flow charts, theprocesses are carried out in parallel to each other at a predeterminedrate and, as a result, data of a produced image is used for adjustmentby the adjustment processing unit 86 at an appropriate timing. Further,some of the illustrated adjustments may be omitted or changed in order.

The adjustment processing unit 86 first performs initial adjustment ofthe white balance using the first grey region 108 in a picked up imageat an initial stage supplied from the gamma correction unit 90 (S30). Inparticular, average values R_ave, G_ave and B_ave of R values, G valuesand B values of a picture of the first grey region 108 are eachcalculated, and gain values Gain_R, Gain_G and Gain_B for the R values,G values and B values are determined in the following manner.

Gain_R=G_ave/R_ave

Gain_G=1

Gain_B=G_ave/B_ave

The gain processing unit 84 multiplies the R values, G values and Bvalues of the RAW images by the gain values by to obtain a picture of anachromatic color in which average luminances of the color components ofthe first grey region 108 are equal to each other, and as a result, thewhite balance is adjusted. After the adjustment processing unit 86 setsthe determined gain values to the gain processing unit 84, it acquires apicked up image after the adjustment from the gamma correction unit 90and confirms whether the luminance of the picture of the second greyregion 110 from within the picked up image after the adjustment is equalto a target value (S32).

By setting a target value in advance to an intermediate luminance of agrey region and adjusting the exposure time period or the gain valuesuch that the set target value may be obtained, such a situation that aluminance or a color component reaches a maximum gradation and issaturated or the variation range of the gradation decreases is preventedwhichever color the block has or even if an image pickup environmentchanges. It is to be noted that the target value may be a single valueor may have some range. In adjustment, the exposure time period isadjusted first, and part that cannot be adjusted with the exposure timeperiod is adjusted with the gain value. This is because, especiallywhere it is necessary to increase the luminance, also noise is amplifiedas the gain value increases.

In particular, where the luminance of a picture of the second greyregion 110 is not equal to the target value (N at S32), if possible, theexposure time period in the image pickup unit 80 is adjusted (Y at S34and S36). In particular, if the luminance is equal to or higher than thetarget value, then the exposure time period is shortened, but if theluminance is lower than the target value, then the exposure time periodis increased. On the other hand, if the exposure time period reaches thelower limit value or the upper limit value of the variation range andcannot be decreased or increased any more (N at S34), then the gainvalue to be set to the gain processing unit 84 is adjusted (S38). Inother words, if the luminance is higher than the target value, then thegain value is decreased, but if the luminance is lower than the targetvalue, then the gain value is increased. It is to be noted that thevariation range of the exposure time period is a range within which anappropriate image is obtained taking the scanning speed, frame rate orthe like of pixels into consideration.

Not only where the exposure time period is adjusted but also where thegain value is adjusted, it is confirmed whether the luminance of thesecond grey region 110 is equal to the target value on the basis of apicked up image obtained after the adjustment by the predeterminedvariation amount (S32). If the luminance of the second grey region 110is not equal to the target value, then the exposure time period or thegain value is adjusted again (S34 and S36 or S38). This is repeateduntil after the luminance of the second grey region becomes equal to thetarget value, and if the target value is reached (Y at S32), then thepicked up image at the setting is acquired and it is confirmed whetherthe luminance of the picture of the white region 104 in the picked upimage is equal to or lower than a predetermined value (S40).

In particular, it is confirmed on an actual image whether the whitecolor is saturated to decide whether or not the adjustment based on thesecond grey region at S32 to S38 is appropriate. For example, if it isassumed that the luminance is indicated by 256 gradations, then it isconfirmed whether the luminance of the white region 104 is equal to orlower than 250. If the luminance is higher than the predetermined value(N at S40), then the gain value of the luminance set by the processestill then is reduced by multiplying the gain value by a coefficientlower than 1 (S42). This process is repeated until after the luminanceof the picture of the white region 104 becomes equal to or lower thanthe predetermined value, and if the luminance becomes equal to or lowerthan the predetermined value (Y at S40), then adjustment based on theluminance difference between the regions is performed as illustrated inFIG. 9.

It is to be noted that adjustment for setting the luminance of a pictureof the black region 106 to a value equal to or higher than apredetermined value in the proximity of the lowest gradation similarlyas in the adjustment of setting the luminance of the picture of thewhite region 104 to a value equal to or lower than the predeterminedvalue at S40 and S42. This prevents such a situation that all blocks ofdark colors come to look black as a result of later processing. However,if such a policy is applied that the adjustment process beginning withS32 adjusts the luminance in only one direction such as a direction inwhich the luminance is gradually increased from a low luminance or inanother direction in which the luminance is gradually decreased from ahigh luminance, then either it is confirmed from the picture of thewhite region 104 whether the gain value is excessively high or it isconfirmed from the picture of the black region 106 whether the gainvalue is excessively low.

Thereafter, the adjustment processing unit 86 acquires picked up imagesaccording to the settings performed till then as depicted in FIG. 9,determines average values in luminance of the pictures of the first greyregion 108 and the second grey region 110 and confirms whether thedifference between the average values reaches a target value (S44). Forexample, if it is intended to set the difference in luminance betweenthe first grey region 108 and the second grey region 110 to 10% of themaximum gradation, a difference in luminance of 25 gradations among 256gradations is required.

The value for which 10% is exemplified here may be a value that reflectsa luminance difference between original colors of the first grey region108 and the second grey region 110 or may be set to a luminancedifference which can be obtained on the image even if the originalcolors do not have such a difference as described above. For example, ifit is intended to handle blocks having colors of luminances close toeach other at the same time, then confusion of the blocks can beprevented by increasing the difference in luminance between picturesfrom that between the original colors. If the luminance differencebetween the pictures of the first grey region 108 and the second greyregion 110 does not reach the target value (N at S44), then thecorrection rule in the gamma correction unit 90 is adjusted such thatthe luminance difference reaches the target value (S46).

Basically, a corresponding portion of the original gamma curve isdeformed. However, actually a different correction formula may beinserted into the portion in place of deforming a line as a graph.Alternatively, a plurality of tables that discretely representcorresponding relationships between the input luminance and the outputluminance may be prepared such that they are selectively utilized.Alternatively else, end points of the pertaining portion may begradually moved such that the portion is interpolated by a linear lineor a curved line such that the original gamma curve connects to theportion. The processes at S44 and S46 are repeated until after theluminance difference between the pictures of the first grey region 108and the second grey region 110 reaches the target value, and if thetarget value is reached (Y at S44), then the adjustment processing unit86 acquires the picked up images at the settings performed till then andadjusts the white balance in each of a plurality of regions thatconfigure the calibration chart 102 from within the picked up images(S48).

This process is basically similar to that at S30, and gain valuesGain_R, Gain_G and Gain_B for the R values, G values and B values in theregions are determined using the formula given hereinabove. Inparticular, since a set of gain values are determined for a luminancerepresented by a picture in each region, a final gain value is set byaveraging for each color component. By the adjustment process describedabove, block recognition having robustness against a variation of anexternal factor becomes possible.

FIG. 10 is a view illustrating white balance adjustment performed at S30of FIG. 8 or S48 of FIG. 9. The upper side in FIG. 10 indicates a colorhistogram and the lower side in FIG. 10 indicates a color histogram ofpictures of the same grey region before adjustment and after adjustment,respectively, and the axis of abscissa indicates the pixel value and theaxis of ordinate indicates the pixel number. Before adjustment, ahistogram 120 for green components, a histogram 122 for blue componentsand histogram 124 for red components represent distributions differentfrom each other. Therefore, by multiplying the pixel values by ratios ofthe average value of G values to the average value of R values and theaverage value of B values as gains for the R values and the B values,respectively, distributions of R values and B values almost equivalentto the distribution of G values like histograms after adjustment areobtained. However, it is not the gist to limit the adjustment techniquefor the white balance to this, but any of techniques realizedpractically may be adopted.

FIG. 11 illustrates a variation of a pixel value when the white balanceis adjusted for a plurality of regions that configure a calibrationchart at S48 of FIG. 9. It is to be noted that the calibration chart inthis case is configured such that, as depicted at an upper stage(calibration chart 126) in FIG. 11, six regions in which achromaticcolors from black to white are represented by six stages of luminanceare juxtaposed in a horizontal direction. The middle stage and the lowerstage represent, where the axis of abscissa indicates the position andthe axis of ordinate indicates the pixel value, variations of pixelvalues of color components when such a calibration chart 126 asdescribed above is scanned in a horizontal direction from the blackside, and the middle stage indicates the variations before theadjustment and the lower stage indicates the variations after theadjustment.

Before the adjustment, a pixel value 128 of a green component, a pixelvalue 130 of a blue component and a pixel value 132 of a red componentindicate values different from each other in all regions. Therefore, byadjusting the white balance in each region to optimize gain values asdescribed above, it is possible to perform adjustment such that thecolor components have pixel values substantially equal to each other inall regions as illustrated in the graph after the adjustment. It is tobe noted that the luminance difference that is adjusted so as to reach atarget value at S44 and S46 of FIG. 9 corresponds, in the case of FIG.11, to the difference D in pixel value, for example, between tworegions.

FIG. 12 is a view illustrating a process for adjusting the exposure timeperiod and the gain value at S32 to S38 of FIG. 8. In the illustration,it is assumed that the demosaic unit 88 generates YCbCr images from RAWimages and uses the Y image (luminance image) among the YCbCr images toperform adjustment for adjusting a luminance value of a picture of thegrey region to a target value. In this case, the luminance Y of eachpixel of RGB images obtained by interpolation of the RAW images iscalculated in accordance with the following expression.

Y=0.299R+0.587G+0.114B

It is assumed that, at S32 of FIG. 8, the luminance of the picture ofthe second grey region before the adjustment has such a value asindicated by “Y” at an upper stage at the left side in FIG. 12. Here, ina rectangle depicted above “Y,” a maximum gradation the luminance valuecan assume is represented by the top side, and the ratio of the actualaverage value of the luminance to the maximum gradation is indicated bya dotted region. Further, in FIG. 12, also the ratios of average valuesof RGB to the maximum gradation are indicated in a similar form at thelower stage. It is to be noted that, since the picture of the secondgrey region is of an achromatic color, where the white balance isadjusted, the average values of RGB become values substantially equal toeach other. Further, in operation, the RGB values vary depending uponthe color of a block.

If the exposure time periods or the gain values when such luminancevalues are obtained are adopted as they are, in regard to a block of acolor having a luminance value higher than that of the second greyregion, the possibility that the luminance of the picture may besaturated is high. Further, in the case of a color whose specificcomponent such as the G value is higher than the others as depicted inFIG. 12, the specific component becomes liable to be saturated, and as aresult, the actual color and the color on the image may become differentfrom each other. Therefore, an appropriate target value is provided forthe luminance of the second grey region, and the exposure time periodand the gain value are adjusted such that such a picture may be obtainedas described hereinabove.

Although a target value is determined strictly depending upon theoriginal luminance of a grey region of an adjustment target, thedetermination is performed qualitatively such that the luminance and theRGB values may not be saturated in regard to any color. For example, ifthe target value is set to 60% of the maximum gradation, then when the Rvalue or the B value is zero, the G value becomes higher than 100% fromthe expression given hereinabove. In other words, the possibility that,even if luminances are similar to each other, saturation may occur witha greenish color is high. If the target value is set to 50%, then wherethe R value or the B value is 0, the G value is 86%. However, thepossibility that the luminance may be saturated with a color having ahigher luminance is high. If the target value is set to 30%, then evenif the R value or the B value is 0, the G value becomes 57% and a marginof 43% occurs.

For example, if input values of RGB are set to 1024 gradations by 10bits, then the G value is maximum at the 1024×57%=583rd gradation whenthe luminance is similar to that of the second grey region. Even whereall of the pixel values of RGB are equal to each other, the RGB valuescan be represented by the 1024×30%=307th gradation. Accordingly, thetarget value is set to approximately 30% as indicated at the upper sideat the right side in FIG. 12, and the exposure time period and the gainvalue are adjusted with the target value. At this time, the exposuretime period is adjusted preferentially as described hereinabove, and ifthe target value is not reached even if the exposure time period reachesthe variation limit, then the gain value of each color component isadjusted.

It is to be noted that to adjust a luminance value as indicated in FIG.12 is nothing but that RGB images are adjusted at an equal ratio.Accordingly, in some cases, target values may be provided for RGB valuessuch that the exposure time period and so forth are adjusted on thebasis of RGB images. In this case, by adjusting the white balance first,target values to be set for the RGG images can be unified to 30% or thelike.

FIG. 13 is a view illustrating a process for adjusting a correction rulein the gamma correction unit 90 at S44 and S46 of FIG. 9. In a generalcamera, gamma correction is performed in order to obtain a good colorbalance on a display unit from output values from image pickup elements.Where a luminance value before correction is an input luminance Y_(in)and a luminance value after correction is an output luminance Y_(out),the gamma correction is executed in accordance with the followingexpression.

Y _(out) =Y _(in) ^((1/γ))  [Expression 1]

Here, γ is a value determined on the basis of characteristics and soforth of a supposed display unit. In FIG. 13, the correction expressionis indicated by a gamma curve 140 of a thin line where the axis ofabscissa indicates the input luminance Y_(in) and the axis of ordinateindicates the output luminance Y_(out). Here, it is assumed that theaverage values in luminance of the pictures of the first grey region 108and the second grey region 110 are Y1 and Y2, respectively. If theaverage values Y1 and Y2 are corrected with the gamma curve 140, thenthe difference D between the output luminances Y_(out) sometimes becomescomparatively small as indicated by blank circles 142 a and 142 b.Therefore, the difference D is compared with the target value, and ifthe difference D does not reach the target value, the gamma curve of thecorrection rule is deformed to expand the difference between the outputluminances Y_(out).

In the example of FIG. 13, a new correction curve 146 (thick line)obtained by deforming a portion of the original gamma curve 140 overwhich the input luminance changes from Y2 to Y1 and a portion in theproximity of the portion of the original gamma curve 140 is generated toexpand the difference D′ between the output luminances Y_(out).Actually, the output value at the blank circle 142 a corresponding tothe input luminance Y1 is increased to that at a black circle 144 a, andthe output value at the blank circle 142 b corresponding to the inputluminance Y2 is decreased to that of a black circle 144 b. The extent ofsuch increase and decrease may be a fixed value or may be varied inresponse to the input luminance. Then, the correction curve 146 isobtained by such interpolation by a general technique that thecorrection curve 146 passes the black circles 144 a and 144 b andconnects to the original gamma curve 140. It is to be noted that thecorrection curve 146 is not limited to a curved line but may partly orentirely be a linear line.

In either way, by expanding the difference between the output luminancesY_(out) from D to D′, distinction in color between blocks having theluminances of the first grey region 108 and the second grey region 110can be recognized with high accuracy. That distinction in color can berecognized with high accuracy signifies that, even if the luminance ofblocks originally having colors same as each other disperses from someexternal factor, the colors can be regarded as the same color. As aresult, block recognition based on a color can be improved in accuracy.

From such a point of view, it is desirable to determine the luminance ofthe first grey region 108 and the second grey region 110 on the basis ofa luminance distribution of colors of blocks used in operation. Forexample, where it is supposed to use two blocks of different colorshaving luminances close to each other, if the luminances of the firstgrey region 108 and the second grey region 110 are adjusted to theluminances of the two colors and then adjustment of the correction curveis performed in such a manner as depicted in FIG. 13, then thedifference between the output values of the luminances of the picturesof them can be emphasized.

Where luminances of three or more colors are close to each other, if thevariation of the luminance of the grey region is increased in accordancewith the luminance values of the blocks like a first grey region, asecond grey region, a third grey region, . . . , then the difference inoutput luminance among pictures of the regions can be emphasized by asimilar technique. Although the foregoing technique adjusts a correctioncurve utilizing a grey region, a correction curve may be adjusted bydirectly evaluating an actual color distribution of blocks.

FIG. 14 is a view illustrating a technique for adjusting a correctioncurve on the basis of a color distribution of blocks. The upper stageindicates a histogram that represents, for each luminance of a color ofa block, a detection frequency of the block detected when a user places,in operation, the block into the space above the play field 100. Inparticular, the distribution is updated in response to a situation inoperation. As described hereinabove, every time the informationprocessing apparatus 10 detects a block newly, it notifies the imagepickup apparatus 12 of color information of the block. The adjustmentprocessing unit 86 of the image pickup apparatus 12 cumulatively recordssuch information to produce and update such a histogram as depicted inFIG. 14. Then, the correction curve is adjusted as occasion demands onthe basis of the histogram.

In the example illustrated in FIG. 14, blocks of colors one-sided to aluminance zone A and another luminance zone B are used. Since colorsbelonging to each luminance zone have similar luminances, thepossibility that they may be confused in operation is high. Therefore, agamma curve is adjusted similarly as in the case illustrated in FIG. 13such that a sufficient difference may appear between the outputluminances. In particular, a difference D_(A) in output luminance whenthe lowest luminance Amin and the highest luminance Amax of theluminance zone A are corrected with the gamma curve and a differenceD_(B) in output luminance when the lowest luminance Bmin and the highestluminance Bmax of the luminance zone B are corrected with the gammacurve are confirmed, and if the differences D_(A) and D_(B) do not reachthe respective target values, then the curve is deformed so as to expandthe differences.

Although, in the case of FIG. 14, the difference D_(A) in outputluminance of the luminance zone A reaches the target value, thedifference D_(B) in output luminance of the luminance zone B does notreach the target value, and such a correction curve as indicated by athick line is produced so as to expand the difference D_(B) to D_(B)′.Consequently, the possibility that colors belonging to the luminancezone B may be confused is suppressed and all blocks can be recognizedwith high accuracy. It is to be noted that, since a frequency equal toor higher than 1 is obtained in the luminance zone B at three luminancesteps, such adjustment may be performed which increases differencesamong three points including an intermediate luminance in addition tothe lowest luminance Bmin and the highest luminance Bmax. Alternatively,the target value to be provided to the difference in output luminancemay be changed in response to the frequency. In the example of FIG. 14,since the frequency of the luminance zone B is high in comparison withthe luminance zone A, the target value for the luminance zone B may beset to a higher value.

Similar adjustment can be performed even if the luminance of a greyregion of the calibration chart 102 is adjusted to the luminance of thecolor of a block supposed to be used instead of the user to adjust acorrection curve on the basis of a picture of a block actually enteredon the play field 100. In particular, such luminance distributions ofblocks as depicted in FIG. 14 are acquired in advance, and four greyregions having the lowest luminance Amin and the highest luminance Amaxof the luminance zone A and the lowest luminance Bmin and the highestluminance Bmax of the luminance zone B are prepared in the calibrationchart 102.

Not only the lowest luminances and the highest luminances but also greyregions of intermediate luminances may be prepared further. Further, ifthe difference D between the output luminances when the luminance of apicture of a grey region is gamma corrected does not reach a targetvalue as depicted in FIG. 13, then the gamma curve is deformed. In thiscase, such a mode that a plurality of blocks of different colors and aplay mat 18 including a calibration chart 102 configured in accordancewith the color distribution of the blocks are provided in a set. It isto be noted that, even if a large number of grey regions between whichthe step size of the luminance is small and a grey region correspondingto the luminance of the color of a block is selected and used foradjustment, a similar function can be implemented. In this case, thegrey region to be used for adjustment is recorded in advance on thebasis of colors of blocks detected in the past or colors of blocks to besupplied in a set.

Part of the calibration chart 102 may be configured from a color chartof chromatic colors confirming to colors of blocks. In this case, thevariable that is the luminance in FIG. 14 is a three-dimensional ortwo-dimensional variable such as (R, G, B) or (R/G, B/G). Further, acolor region in which colors of blocks are dense in comparison with apredetermined reference density in the color space is detected, and if astandard deviation after correction of the colors or a distance betweencolors in the color space does not reach a target value, then the gammacurve is deformed so as to expand the standard deviation or thedistance.

A difference may be provided between colors before gamma correction byutilizing this color chart to adjust the gain value for each colorcomponent by the gain processing unit 84. FIG. 15 is a view illustratinga technique for adjusting the gain values of RGB utilizing the colorchart. This process is performed, for example, in the flow chart of FIG.8, after the processes at S32 to S42 by which the exposure time periodor the gain value is adjusted on the basis of a luminance of a pictureof a grey region of the calibration chart 102. First, it is assumedthat, as the colors of blocks, a first color, a second color and a thirdcolor similar to each other are available. The left side in FIG. 15 is acolor histogram of each color, which is represented in a form similar tothat of FIG. 10.

In this example, it is assumed that a region of a color same as thesecond color (the region is hereinafter referred to as second colorregion) is prepared on the calibration chart 102. Then, after theprocesses at S32 to S42 described hereinabove, the adjustment processingunit 86 extracts a picture of the second color region in a picked upimage of the calibration chart 102 and adjusts the white balance withthe picture. In other words, the gain values of the color components aredetermined such that the second color becomes an achromatic color. Themanner of adjustment of the white balance is similar to that describedhereinabove with reference to FIG. 10. Consequently, distributions ofthe color components become substantially equal to each other as in acolor histogram of the second color at the right side in FIG. 15.

If an image of actual blocks is picked up with the determined gainvalues, then the histograms of the first color and the third colorbecome such as those depicted at the right side. Although this colorinformation is different from the original colors of the first color,second color and third color, a difference among the first to thirdcolors appears clearly. For example, although, in the original color,the first to third colors are all greenish colors having high G values,after the adjustment, the first color is a strongly bluish color and thesecond color is close to an achromatic color while the third color is astrongly reddish color.

By preparing a color chart of at least one of colors of blocks to beused and adjusting the white balance with the color in this manner,distinction of the color from similar colors is emphasized in comparisonwith the original color. As a result, even if color information acquiredfrom a picked up image has some dispersion, confusion with any othercolor can be avoided. Where a plurality of similar color sets exist likethe luminance zones depicted in FIG. 14, a color chart of an averagecolor or the like may be prepared for each color set.

FIG. 16 depicts a modification to the play mat in the presentembodiment. A play mat 150 of the present example is configured suchthat a play field 152 is provided over a substantially overall area anda calibration chart 154 is prepared separately. Further, when an imagepickup condition or a correction condition of an image is to be adjustedusing the calibration chart 154, the calibration chart 154 is placed onand used together with the play field 152. The calibration chart 154 maybe stuck by a hook and loop fastener or the like. On the other hand, inoperation where a block is used, the calibration chart 154 is removed toexpose the play field 152.

Where the play mat is configured in such a manner as described above,both of the play field 152 and the calibration chart 154 can beconfigured so as to have an area equal to that of the play mat 150. As aresult, a greater number of regions of different luminances or colorscan be prepared as the calibration chart 154, and blocks can be handledat a place of a greater area. Further, even if the field of view of theimage pickup apparatus 12 is fixed, an image of both of the calibrationchart 154 and the play field 152 can be picked up well. In place ofconfiguring the calibration chart 154 so as to be removably mounted inthis manner, the play mat may be configured such that a calibrationchart is indicated over a substantially overall area of the rear face ofthe play mat 150. In this case, if the user turns over the play mat asoccasion demands, then an effect similar to that described hereinabovecan be achieved.

According to the present embodiment described above, in an informationprocessing system including block recognition based on a color, acalibration chart including a plurality of regions of differentluminances or colors is provided on a play mat that defines a targetspace from which a block is to be detected. Then, as an initial processin an image pickup apparatus, at least one of an exposure time period, again value to be provided to each color component and a correction ruleupon gamma correction is adjusted on the basis of a picture of thecalibration chart in a picked up image.

In particular, the exposure time period and the gain value to beprovided to each color component are adjusted on the basis of theluminance of a picture of a grey region such that a gradation number canbe utilized utmost within a range within which the value of a luminanceor a color component is not saturated with any color. Further, the whitebalance is adjusted on the basis of a region of an achromatic color.Furthermore, a correction curve upon gamma correction is adjusted suchthat an output luminance difference between regions of differentluminances may appear sufficiently.

Further, by providing a region of a color same as that of a block on acalibration chart and adjusting the white balance of the same, a colorimage in which the difference between the color and a similar color isemphasized is produced. In addition to the calibration chart, acorrection curve upon gamma correction may be adjusted in response to acolor distribution or a luminance distribution of a block actuallyentering a space above a play field such that the difference inluminance or color in an output image is emphasized. From those, colorinformation of a block can be acquired from a picked up image in a statein which the influence of a variation of the image pickup environmentsuch as illumination or presence or absence of a different article.Further, confusion between blocks having similar colors can be avoided.As a result, the recognition accuracy of a block based on a color can beimproved.

The present invention has been described based on the embodiment. Theembodiment described above is exemplary, and it is recognized by thoseskilled in the art that various modifications are possible incombination of the components or the processes in the embodiment andthat also such modifications are included within the scope of thepresent invention.

REFERENCE SIGNS LIST

1 Information processing system, 20 a Block, 10 Information processingapparatus, 12 Image pickup apparatus, 14 Inputting apparatus, 16 Displayapparatus, 18 Play mat, 22 CPU, 24 GPU, 26 Main memory, 52 Inputinformation acquisition unit, 54 Picked up image acquisition unit, 60Detection unit, 62 Recognition unit, 64 Block database, 70 Output dataproduction unit, 72 Image data storage unit, 80 Image pickup unit, 84Gain processing unit, 86 Adjustment processing unit, 88 Demosaic unit,90 Gamma correction unit, 92 Data sending out unit, 100 Play field, 102Calibration chart, 150 Play mat, 152 Play field, 154 Calibration chart.

INDUSTRIAL APPLICABILITY

As described above, the present invention can be utilized in toys,learning equipment, computers, game apparatus, and informationprocessing apparatus, and systems including them.

1. An image pickup apparatus that outputs data of a picked up image tobe used for recognition of a real object, comprising: an output imageproduction unit configured to produce data of a picked up image to beoutputted from data of a non-processed image picked up using an imagepickup element; an adjustment processing unit configured to evaluate thepicked up image and adjust at least one of an image pickup condition inthe image pickup element and a processing condition in the output imageproduction unit; and a data sending out unit configured to output thedata of the picked up image to an apparatus that recognizes the realobject, wherein the adjustment processing unit adjusts the image pickupcondition and the processing condition based on color information of apicture of a calibration chart that is provided in a space of arecognition target and is configured from a plurality of color regionshaving luminances different from each other.
 2. The image pickupapparatus according to claim 1, wherein the adjustment processing unitevaluates the luminance of a picture of the color regions of thecalibration chart and adjusts the processing condition so as to expandthe difference between a plurality of luminances corresponding to theluminance of the color of the real object.
 3. The image pickup apparatusaccording to claim 2, wherein the adjustment processing unit determinesa luminance range within which the difference is to be expanded based ona luminance distribution of the picture of the real object in the pickedup image.
 4. The image pickup apparatus according to claim 1, whereinthe adjustment processing unit evaluates a color of a picture of a colorregion having a color same as a color of the real object in thecalibration chart and adjusts the processing condition so as to expand adistance between a plurality of colors of different pictures in a colorspace.
 5. The image pickup apparatus according to claim 2, wherein theadjustment processing unit adjusts, as the adjustment of the processingcondition, a correction rule upon gamma correction.
 6. The image pickupapparatus according to claim 1, wherein the adjustment processing unitadjusts a gain value of each color component such that a picture of acolor region in the calibration chart, which has a color same as one ofchromatic colors of the real object, has an achromatic color.
 7. Theimage pickup apparatus according to claim 1, wherein the adjustmentprocessing unit evaluates a luminance of a picture of a color region inthe calibration chart, which has a given luminance, and adjusts theimage pickup condition and the processing condition in a direction inwhich the luminance is adjusted to a target luminance determined inadvance.
 8. The image pickup apparatus according to claim 1, wherein theadjustment processing unit adjusts a white balance to a picture of acolor region of an achromatic color in the calibration chart.
 9. Theimage pickup apparatus according to claim 1, wherein the data sendingout unit parallelly outputs data of a picked up image produced inaccordance with a processing condition adjusted by the adjustmentprocessing unit and data of a picked up image produced in accordancewith a processing condition that omits at least part of the adjustmentby the adjustment processing unit.
 10. The image pickup apparatusaccording to claim 1, wherein the data sending out unit switches andoutputs data of a picked up image produced in accordance with aprocessing condition adjusted by the adjustment processing unit and dataof a picked up image produced in accordance with a processing conditionthat omits at least part of the adjustment by the adjustment processingunit in accordance with a request from an apparatus that recognizes thereal object.
 11. An information processing system including an imagepickup apparatus that outputs data of a picked up image obtained byimage pickup of a real object and an information processing apparatusthat recognizes the real object using the picked up image, wherein theimage pickup apparatus includes an output image production unitconfigured to produce data of a picked up image to be outputted fromdata of a non-processed image picked up using an image pickup element,an adjustment processing unit configured to evaluate the picked up imageand adjust at least one of an image pickup condition in the image pickupelement and a processing condition in the output image production unit,and a data sending out unit configured to output the data of the pickedup image to the information processing apparatus, and the adjustmentprocessing unit adjusts the image pickup condition and the processingcondition based on color information of a picture of a calibration chartthat is provided in a space of a recognition target and is configuredfrom a plurality of color regions having luminances different from eachother, and the information processing apparatus recognizes a block bycomparison between color information of the picture of the real objectin the picked up image and color information of real objects registeredalready with each other and performs a process set for the block.
 12. Amat that is placed in an image pickup space in an information processingsystem that recognizes a real object based on color information of apicture in a picked up image, comprising: a play field that defines abottom region of a space in which, when the real object is placed intothe space, the real object is determined as a recognition target; and acalibration chart configured from a plurality of color regions havingdifferent luminances from each other and used by an image pickupapparatus to adjust, based on color information of a picture in thepicked up image, at least one of an image pickup condition and aprocessing condition when data of the picked up image to be outputted isproduced.
 13. The mat according to claim 12, wherein the calibrationchart includes a color region having a luminance corresponding to asupposed color of the real object of the recognition target.
 14. The mataccording to claim 12, wherein the mat includes a first mat thatincludes the play field and a second mat that includes the calibrationchart, and the second mat is placed and removably mounted on the firstmat.
 15. An image production method, by an image pickup apparatus, ofproducing data of a picked up image to be used for recognition of a realobject, comprising: producing data of a picked up image to be outputtedfrom data of a non-processed image picked up using an image pickupelement; evaluating the picked up image and adjusting at least one of animage pickup condition in the image pickup element and a processingcondition in the producing; and outputting the data of the picked upimage to an apparatus that recognizes the real object, wherein theadjusting adjusts the image pickup condition and the processingcondition based on color information of a picture of a calibration chartthat is provided in a space of a recognition target and is configuredfrom a plurality of color regions having luminances different from eachother.
 16. A non-transitory, computer readable storage medium containinga computer program, which when executed by a computer, causes thecomputer to carry out actions to implement a function for producing dataof a picked up image to be used for recognition of a real object, thecomputer program causing the computer to carry out the actions of:producing data of a picked up image to be outputted from data of anon-processed image picked up using an image pickup element; evaluatingthe picked up image and adjusting at least one of an image pickupcondition in the image pickup element and a processing condition in theproduction function; and outputting the data of the picked up image toan apparatus that recognizes the real object, wherein the adjustingadjusts the image pickup condition and the processing condition based oncolor information of a picture of a calibration chart that is providedin a space of a recognition target and is configured from a plurality ofcolor regions having luminances different from each other. 17.(canceled)